fix: scalar timestamp assignment (#19843) #19973

DylanDmitri · 2018-03-02T20:36:25Z

closes BUG: scalar assignment of a tz-aware is object dtype #19843
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

gfyoung · 2018-03-02T21:03:31Z

pandas/tests/scalar/timestamp/test_assignment_dtype.py

+from pandas.core.dtypes.dtypes import DatetimeTZDtype
+
+
+# referencing #19843: scalar assignment of a tz-aware is object dtype


I think we can find a home for this in another test file (and don't need a new one). In addition, the comment should be under the class declaration.

@jreback : I think tests/frame/test_indexing.py makes sense here. What do you think?

Sounds good, should they go under TestDataFrameIndexingDatetimeWithTZ, or in their own class?

Same class sounds good for now.

gfyoung · 2018-03-02T21:48:59Z

pandas/tests/frame/test_indexing.py

@@ -3042,6 +3044,22 @@ def test_transpose(self):
        expected.index = ['A', 'B']
        assert_frame_equal(result, expected)

+    def test_scalar_assignment(self):


Reference issue number for both tests.

DylanDmitri · 2018-03-02T23:12:22Z

what should I do for the whatsnew?

gfyoung · 2018-03-03T00:14:11Z

Perhaps something like:

Fixed `DataFrame` assignment with a timezone-aware object (:issue:`19843`)

codecov · 2018-03-03T03:19:13Z

Codecov Report

Merging #19973 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #19973      +/-   ##
==========================================
+ Coverage   92.06%   92.06%   +<.01%     
==========================================
  Files         169      169              
  Lines       50681    50690       +9     
==========================================
+ Hits        46659    46668       +9     
  Misses       4022     4022

Flag	Coverage Δ
#multiple	`90.47% <100%> (ø)`	⬆️
#single	`42.32% <100%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/frame.py	`97.26% <100%> (ø)`	⬆️
pandas/io/parsers.py	`95.48% <0%> (+0.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf14627...1b9649c. Read the comment docs.

jreback · 2018-03-04T19:47:29Z

doc/source/whatsnew/v0.23.0.txt

@@ -854,6 +854,7 @@ Datetimelike
 - Bug in :func:`to_datetime` where passing an out-of-bounds datetime with ``errors='coerce'`` and ``utc=True`` would raise ``OutOfBoundsDatetime`` instead of parsing to ``NaT`` (:issue:`19612`)
 - Bug in :class:`DatetimeIndex` and :class:`TimedeltaIndex` addition and subtraction where name of the returned object was not always set consistently. (:issue:`19744`)
 - Bug in :class:`DatetimeIndex` and :class:`TimedeltaIndex` addition and subtraction where operations with numpy arrays raised ``TypeError`` (:issue:`19847`)
+- Bug in `DataFrame` assignment with a timezone-aware object (:issue:`19843`)


use double-backticks are DataFrame

say a timezone-aware scalar

jreback · 2018-03-04T19:47:51Z

pandas/core/frame.py

@@ -2868,9 +2868,15 @@ def reindexer(value):
                value = maybe_infer_to_datetimelike(value)

        else:
-            # upcast the scalar
+            # cast ignores pandas dtypes. so save the dtype first
+            from pandas.core.dtypes.cast import infer_dtype_from_scalar


import at the top of the file with other imports

jreback · 2018-03-04T19:48:01Z

pandas/core/frame.py

-            # upcast the scalar
+            # cast ignores pandas dtypes. so save the dtype first
+            from pandas.core.dtypes.cast import infer_dtype_from_scalar
+            pd_dtype, _ = infer_dtype_from_scalar(value, pandas_dtype=True)


call the return dtype

jreback · 2018-03-04T19:48:22Z

pandas/core/frame.py

            value = cast_scalar_to_array(len(self.index), value)
-            value = maybe_cast_to_datetime(value, value.dtype)
+


you don't need the blank line and the comment here

jreback · 2018-03-04T19:48:44Z

pandas/tests/frame/test_indexing.py

+        # issue #19843
+        df = pd.DataFrame(index=(0, 1, 2))
+        df['now'] = pd.Timestamp('20130101', tz='UTC')
+        assert isinstance(df.dtypes[0], DatetimeTZDtype)


construct the expected frame and use assert_frame_equal

put this in class TestDataFrameIndexingDatetimeWithTZ(TestData)

name it test_setitem_scalar

parameterize it with both a tz and a tz-aware value.

jreback · 2018-03-04T19:50:04Z

pandas/tests/frame/test_indexing.py

+        df['now'] = pd.Timestamp('20130101', tz='UTC')
+        assert isinstance(df.dtypes[0], DatetimeTZDtype)
+
+    def test_datetime_index_assignment(self):


this is a duplicate test.

you could put it as an additional test in the class as above if you'd like (parametrize as well)

jreback · 2018-04-14T14:37:49Z

can you rebase and update

jreback · 2018-07-28T14:36:08Z

can you rebase & move the note to 0.24.0

jreback · 2018-08-02T11:00:34Z

fixed up, but going to do in ~~0.23.4.~~ 0.24.0

* master: (47 commits) Run tests in conda build [ci skip] (pandas-dev#22190) TST: Check DatetimeIndex.drop on DST boundary (pandas-dev#22165) CI: Fix Travis failures due to lint.sh on pandas/core/strings.py (pandas-dev#22184) Documentation: typo fixes in MultiIndex / Advanced Indexing (pandas-dev#22179) DOC: added .join to 'see also' in Series.str.cat (pandas-dev#22175) DOC: updated Series.str.contains see also section (pandas-dev#22176) 0.23.4 whatsnew (pandas-dev#22177) fix: scalar timestamp assignment (pandas-dev#19843) (pandas-dev#19973) BUG: Fix get dummies unicode error (pandas-dev#22131) Fixed py36-only syntax [ci skip] (pandas-dev#22167) DEPR: pd.read_table (pandas-dev#21954) DEPR: Removing previously deprecated datetools module (pandas-dev#6581) (pandas-dev#19119) BUG: Matplotlib scatter datetime (pandas-dev#22039) CLN: Use public method to capture UTC offsets (pandas-dev#22164) implement tslibs/src to make tslibs self-contained (pandas-dev#22152) Fix categorical from codes nan 21767 (pandas-dev#21775) BUG: Better handling of invalid na_option argument for groupby.rank(pandas-dev#22124) (pandas-dev#22125) use memoryviews instead of ndarrays (pandas-dev#22147) Remove depr. warning in SeriesGroupBy.count (pandas-dev#22155) API: Default to_* methods to compression='infer' (pandas-dev#22011) ...

fix: scalar timestamp assignment (pandas-dev#19843)

5a9b1b9

gfyoung added Datetime Datetime data dtype Dtype Conversions Unexpected or buggy dtype conversions Timedelta Timedelta data type Timezones Timezone data dtype and removed Timedelta Timedelta data type labels Mar 2, 2018

gfyoung reviewed Mar 2, 2018

View reviewed changes

moved tests

601cf54

gfyoung reviewed Mar 2, 2018

View reviewed changes

issue number in tests

33f8651

added whatsnew

59069b9

jreback requested changes Mar 4, 2018

View reviewed changes

jreback added 3 commits August 2, 2018 06:50

Merge branch 'master' into PR_TOOL_MERGE_PR_19973

edcb4e2

update docs

c48fbe9

clean

1b9649c

jreback modified the milestones: 0.24.0, 0.23.4 Aug 2, 2018

jreback approved these changes Aug 2, 2018

View reviewed changes

jreback modified the milestones: 0.23.4, 0.24.0 Aug 2, 2018

jreback merged commit 6e1e1e4 into pandas-dev:master Aug 2, 2018

dberenbaum pushed a commit to dberenbaum/pandas that referenced this pull request Aug 3, 2018

fix: scalar timestamp assignment (pandas-dev#19843) (pandas-dev#19973)

fd594cf

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

fix: scalar timestamp assignment (pandas-dev#19843) (pandas-dev#19973)

daa0023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: scalar timestamp assignment (#19843) #19973

fix: scalar timestamp assignment (#19843) #19973

DylanDmitri commented Mar 2, 2018 •

edited

Loading

gfyoung Mar 2, 2018

DylanDmitri Mar 2, 2018

gfyoung Mar 2, 2018

gfyoung Mar 2, 2018

DylanDmitri commented Mar 2, 2018

gfyoung commented Mar 3, 2018

codecov bot commented Mar 3, 2018 •

edited

Loading

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback Mar 4, 2018

jreback commented Apr 14, 2018

jreback commented Jul 28, 2018

jreback commented Aug 2, 2018 •

edited

Loading

		from pandas.core.dtypes.dtypes import DatetimeTZDtype


		# referencing #19843: scalar assignment of a tz-aware is object dtype

		value = cast_scalar_to_array(len(self.index), value)
		value = maybe_cast_to_datetime(value, value.dtype)

fix: scalar timestamp assignment (#19843) #19973

fix: scalar timestamp assignment (#19843) #19973

Conversation

DylanDmitri commented Mar 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DylanDmitri commented Mar 2, 2018

gfyoung commented Mar 3, 2018

codecov bot commented Mar 3, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Apr 14, 2018

jreback commented Jul 28, 2018

jreback commented Aug 2, 2018 • edited Loading

DylanDmitri commented Mar 2, 2018 •

edited

Loading

codecov bot commented Mar 3, 2018 •

edited

Loading

jreback commented Aug 2, 2018 •

edited

Loading